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REAL TIME AUDIO SPATIALISATION SYSTEM WITH fflGH 

LEVEL CONTROLS 

Description 

5 The present invention relates to a system by which a hstener (user) can 

control the spatialization of sound sources (also called «tracks») in real time, 
to produce a spatialized mixing (so-called multi-channel sound) which satisfy 
a set of so-called «constraints», defined a priori, and stored in the audio file. 
As an audio version, MusicSpace provides a high level command 

10 language for moving groups of related sound sources, and may be used to 
control arbitrary spatialization systems. MusicSpace was connected 
successfully to a Midi Spatialization system for playing midi files, to a midi- 
controlled audio mixing console for mixing multi-track recordings, as well as 
to Lrcam's spatialization system (Jot, J,-M. and Warusfel, O,, "A Real-Time 

15 Spatial Sound Processor for Music and Virtual ReaUty Applications", 
Proceedings of ICMC, 1995. These authors also filed several patent 
applications about sound spatialization). In all these cases though, 
MusicSpace is used as a control system, not as a spatialization system per se. 
There was recently built a specific Dynamic Link Library (dll) for 

20 Window-based PCs which allows MusicSpace to control Microsoft DirectX 
3D sound buffers. This dll of MusicSpace-audio basically provides a 
connection between any Java application and DirectX, by converting DirectX 
*s API types into simple types (such as integers) that can be handled by 
Java. 

25 In order to achieve the above-mentioned object, there is provided a 

method of spatializing sound sources so as to produce a spatialized mixing in 
real time, the method comprising the steps of: 
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a) providing an audio carrier including music titles, each of the music 
titles having a set of specific constraints; and 

b) dynamically mixing the sound sources through a decoding module, 
so that the set of constraints are always satisfied. 

5 Typically, the above method further comprises the step of providing the 

audio carrier from individual audio tracks and a database through an encoding 
module. 

In the present invention, there is also provided a system adapted to 
implement the above-described method, the system comprising a general- 
10 purpose computer. 

Although DirectX may arguably not be the more accurate spatialization 
system around, this extension has a number of benefits. 

First, DirectX provides parameters for describing 3D sound sources 
which can be constrained using MusicSpace. For instance, a DirectX sound 
15 source is endowed with an orientation, a directivity and even a Doppler 
parameter. An «orientation» constraint has been designed and included in the 
constraint library of MusicSpace. This constraint states that two sound source 
should always «face» each other: when one source is moved, the orientation 
of the two sources move accordingly. 
20 Second, DirectX allows to handle lots of soimd sources in real time. 

This is useful for mixing complex symphonic music, which have often dozens 
of related sound sources. 

Lastly, the presence of DirectX on a number of PC makes MusicSpace 
easily useable to a wide audience. 
25 More precisely, the invention allows to: 

• Listen to music recorded on a single audio support such as DVD, CD-ROM 
or Super CD audio, (not exclusive) 
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• Perform dynamic audio mixing, i.e. mixing of audio tracks in real time. 

• Use high level operators to control this mixing, 

• in such a way that a set of «constraints» are always satisfied, this set of 
constraint being specific for each music title. 

5 The present invention is also based on a previous European Patent 

Application No. 98401266.6, entitled «Mixer using constraint propagation)), 
concerning the underlying constraint technology for relating sound sources 
together. 

The originality of this present invention lies in : 
10 1) Coding and decoding of multiple audio sources from a single support (e.g, 
CD-ROM), and therefore explicit handling of audio sources (the previous 
patent application focused on the Midi format). 
2) Introduction of high level control, on top of the audio source layer. 

The above and other objects, features and advantages will be made 
15 apparent from the following description of the preferred embodiments, given 
as non-limiting examples, with reference to the accompanying drav^ngs, in 
which Fig, 1 illustrates the general data flow of the invention. 

Specifications of modules are summarised as follows: 
1) Audio and metadata encoder 
20 This modules takes as input: 

a) a set of individual audio tracks (monophonic format, all other parameters 
can be acconmiodated by the invention, e.g. sampling rate, resolution, etc.,) 

b) a set of metadata, describing the constraints and related constrained 
variables needed for the constraint system. These metadata are represented in 

25 a symbolic, textual format. 

c) a format name 
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The format name must support multiplexed audio data and arbitrary 
metadata, like: AIFF, WAV, or Mpeg4 (not exclusive). 

According to the format name, the module encodes the audio tracks and 
the metadata into a single file. 

The specification of the format of the metadata is given infra* 
2) Audio and metadata decoder 

This module takes as input a file in one of the formats created by the encoder. 
It recreates: 

a) a set of audio streams from each individual tracks 

b) the specification of the metadata from the encoded format. 

The set of audio streams is given as input to the spatialization module. 
The set of metadata is given as input to the constraint system module. 



3) Constraint System 

15 The constraint system is an extension of the constraint system 

described in previous European Patent Application No. 98401266.6 supra. 
The extension consists in: 

• Introducing new constrained variables, called «handles». These variables 
are just constrained variables which are not assigned to a particular audio 

20 track. They are constrained by standard constraints already described in 
previous patent application, or by one-way constraints, described below. 

• Extending the constraint propagation mechanism to include the 
management of so-called «one-way constraints)). This extension of the 
constraint solver consists in propagating the perturbation in a constraint 

25 «only» in the directions allowed by the constraint. More description to 
come here, including a pseudo-code of one way constraints. 

4) Spatialization controller 
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The spatialization controller module takes as input the following 
information: 

• The set of individual audio streams as decoded by the decoder module. 

• Spatialization commands given by the constraint system. 

5 This module is identical to the module described in the previous patent 

application supra, except that there has been designed a specific version for 
reusing the DirectX spatialization middleware of Microsoft. 

Specifications of the metadata format are as follows, 
each configuration of constraint set is represented by a String as follows: 

10 The format contains two parts: 

• «variable part» 

• «constranit part» 

a) Variable part 

Each individual sound track is given a number from 1 to n. Each track 
15 parameter is specified, one by one, in the following order: 

• track name, 

• track icon, 

• individual loudness 

• initial position (x, y coordinates) 
20 • etc.,, 

b) Constraint part 

Each constraint is represented by the following information: 

• constraint type, 

• constrained variables (numbers of individual tracks) 
25 • constraint position 

• constraint orientations 
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The invention has applications also outside the field of spatialization. 
It can be used for any situation where : 

1) Streams of real time data can be controlled by discrete parameters (e.g. 
5 streams of audio sources controlled by distance, pan, directivity, etc.), 

2) Relations between these parameters can be expressed as constraints or 
combinations of constraints. 

Such situations occur frequentiy in music composition, sound 
synthesis, and real time control. Only some of them were described supra. 
10 Other applications in progress concern the automatic animation of sound 
sources (e.g. defining sources which revolve automatically around other 
sources, or which move through a path itself defined with constraints). 



1 

Claims: 

1. A method of spatializing sound sources so as to produce a spatialized 
mixing in real time, said method comprising the steps of: 

a) providing an audio carrier including music titles, each of said music 
5 titles having a set of specific constraints; and 

b) dynamically mixing said sound sources through a decoding module, 
so that said set of constraints are always satisfied. 

2. The method according to claim 1, said method further comprising the 
step of providing said audio carrier from individual audio tracks and a 

10 database through an encoding module. 

3. A system adapted to implement the method of claim 1 or 2, 
comprising a general-purpose computer. 
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Abstract 

The invention relates to a system in which a user can control the 
spatialization of sound sources in real time, and produce a spatialized mixing. 
In this system, the user listens to a music recorded on a single audio support 
such as DVD or CD-ROM, and performs dynamic audio mixing under high- 
level controls. To enable the above operation, each music title is provided 
with a set of specific constraints, and the dynamic audio mixing must satisfy 
the set of constraints. 
FIG. 1 
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